The present invention relates to building photographic panoramas of virtual reality and in particular to building spherical environmental maps utilizing some photographs.
In virtual reality systems, synthesis and navigation of virtual environments are usually accomplished using one of the following approaches: 1) three-dimensional modeling and rendering; 2) branching movies; and 3) image-based approach.
Although three-dimensional modeling and rendering is the traditional approach, creating the three-dimensional geometrical entities is a laborious manual process, and special hardware (e.g., a graphic accelerator) may be necessary for rendering, in real time, a highly complex scene.
The branching movies approach is used extensively in the video game industry. Multiple movie segments are connected together to depict spatial paths of selected branch points. This approach is limited in navigability and interaction, and also requires a large amount of storage space to store movies.
For rendering complex scenes, such as real-world scenery, the image-based approach is often considered to be the most reasonable and feasible approach. The image-based approach renders scenes within a fixed time as well as independently of rendering quality and scene complexity.
The image based approach uses an environment map, which is established from a collection of images and characterizes the appearance of a scene when viewed from a particular position. The environment map contains the pixel values used to display the scene. The image environment map is first wrapped onto an object surface having a certain geometry, such as a cube, a sphere, or a cylinder. Afterwards, by locating the viewing position at the geometrical center of the wrapped object, perspective-corrected views of scenes can be reconstructed from the image on the object surface during playback.
There are many types of environment maps for storing data including cubic maps, spherical maps, cylindrical maps, fish-eye or hemispherical maps, and planar maps. Some environments are capable of storing data of omni-directional scenes, while others cannot. Others have used two additional kinds of environment maps in experimental systems: 1) the textured cylinder-like prism, and 2) the texture sphere-like polyhedron (see e.g., W. K. Tsao, xe2x80x9cRendering scenes in the real world for virtual environments using scanned imagesxe2x80x9d, Master Thesis of National Taiwan University, Advisor: Ming Ouhyoung, 1996). Rendering methods for different types of environment maps are described in U.S. Pat. Nos. 5,396,583; 5,446,833; 5,561,756; and 5,185,667, all of which are incorporated herein by reference.
As for building image-based panoramas, commercially available software (e.g., the QuickTime VR authoring tool suite, from Apple Computer Inc.) can be used to create a seamless panoramic cylindrical image from a set of overlapping pictures. An improved method of a stitching method is described in Hsieh et al., U.S. Ser. No. 08/933,758, filed Sep. 23, 1997, which is incorporated herein by reference. Stitching together photographic images taken from a fish-eye lens is also described in the developer manual of InfinitePictures(trademark), Inc. (xe2x80x9cSmoothMove Panorama Web Builderxe2x80x9d, Developer Manual (Version 2.0) of InfinitePictures(trademark), Inc., 1996).
Spherical environment maps are used to store data relating to a surrounding scene. There exists many types of spherical environment maps. Spherical mapping mechanisms are described in U.S. Pat. Nos. 5,359,363, 5,384,588, 5,313,306, and 5,185,667, all of which are incorporated herein by reference. The major advantage of spherical mapping systems is that the environment map is able to provide users with 360 degrees of both horizontal and vertical pannings of views within the mapped sphere. However, a common problem with spherical mapping systems relates to image acquisition. Generally, to acquire the image for a spherical environment map, users usually require special and relatively expensive cameras having fish-eye lenses. Two types of particular interest are the xe2x80x9cspherical reflectionxe2x80x9d map and the xe2x80x9cparametric spherical environmentxe2x80x9d map.
A xe2x80x9cspherical reflectionxe2x80x9d map stores the image of an environment as an orthographic projection on a sphere representing a perfect reflection of the surrounding scene (Zimmermann U.S. Pat. No. 5,185,667). These images with circular shapes are all stored in respective square arrays of pixels. The major disadvantage of spherical reflection maps is that the orientations near the silhouette of the sphere (i.e., the ring-like shape when the sphere is orthographically projected on a sphere) are sampled very sparsely. This makes it more difficult to render the spherical reflection maps for omni-directional and interactive viewers.
The invention relates to building spherical panoramas or spherical environment maps for image-based virtual reality systems. The image-based spherical panoramas can be navigated in any desired view direction (360-degrees) by suitable three-dimensional image browsers or viewers.
In one aspect of the invention, a method of photographing pictures for building the spherical panoramas with a camera system includes determining a number of circles of latitude to build the spherical panorama on the basis of the focal length of the camera system, the height of photographic film used in the camera system, and an overlapping ratio between contiguous ones of the pictures, each circle of latitude including a series of contiguous pictures; and photographing the determined number of pictures.
The approach permits the use of a camera having a standard or wide-angled lenses rather than fish-eye lenses which are more expensive and less popular.
Embodiments of this aspect of the invention may include one or more of the following features.
Determining the number of circles of latitude includes calculating the following equation:   NumCirclesLandscape  =            [                                    180            ⁢            xc2x0                                              (                              100                -                k                            )                        ⁢            %            *                          [                              2                ⁢                                                      tan                                          -                      1                                                        ⁡                                      (                                          a                                              2                        ⁢                        f                                                              )                                                              ]                                      -        1            ]        .  
where:
a is the height of the photographic film;
f is the focal length of the camera system; and
k is the overlapping ratio between contiguous pictures.
Alternatively, the number of circles of latitude can be determined by calculating the following equation:   NumCirclesPortrait  =            [                                    180            ⁢            xc2x0                    -                      2            ⁢                                          tan                                  -                  1                                            ⁡                              (                                  b                                      2                    ⁢                    f                                                  )                                              +                      [                          2              ⁢              k              ⁢              %              *                                                tan                                      -                    1                                                  ⁡                                  (                                      a                                          2                      ⁢                      f                                                        )                                                      ]                                                (                          100              -              k                        )                    ⁢          %          *                      [                          2              ⁢                                                tan                                      -                    1                                                  ⁡                                  (                                      a                                          2                      ⁢                      f                                                        )                                                      ]                              ]        .  
where:
a is the height of the photographic film;
b is the width of the photographic film;
f is the focal length of the camera system; and
k is the overlapping ratio between contiguous pictures.
In another aspect of the invention, a method of transforming photographic images of a panoramic scene into a spherical environment map includes warping each of the plurality of photographic images into a parametric spherical environment map; and image cutting each of the warped photographic images into a rectangular image based on an elevation angle of a predetermined sight direction.
This aspect of the invention provides a reliable and efficient approach for transforming photographic images into a spherical environment map. A spherical environment map can be linearly mapped using the spherical coordinate system, in which each pixel is specified in terms of the pixel""s azimuthal angle around the horizon and its elevation angle below the north pole, denoted by (xcex8, xcfx86). Based on angles (xcex8v, xcfx86v) which define a picture""s sight (or view) direction, a two-dimensional image of a picture can be transformed onto an image region of a spherical environment map using an image warping formula. This is regarded as one kind of coordinate transformation. Although each picture is taken based on the predefined sight direction, direction errors in taking pictures may exist. For computing the actual azimuth angles of camera sight direction, an additional coordinate transformation using the azimuthal coordinate system is performed. Using these computed azimuth angles, the picture images are seamed together into a spherical environment map utilizing an image stitching technique.
Embodiments of this aspect of the invention may include one or more of the following features.
During image cutting, the warped photographic images are cut either horizontally or vertically on the basis of the elevation angle of the sight direction for each of the warped photographic images. For example, if the elevation angle of the sight direction, relative to one of the poles, for a warped photographic image is less than 10xc2x0 or greater than 170xc2x0, the warped photographic image is cut into a rectangular region defined by side edges of the warped photographic image and either a bottom or a top edge of the warped photographic image, respectively. On the other hand, if the elevation angle of the sight direction for a warped photographic image is between 30xc2x0 and 150xc2x0, the warped photographic image is cut into a rectangular region defined by two opposing horizontal lines and two opposing vertical lines.
Warping each of the photographic images includes mapping each pixel of the warped photographic image from at least one pixel of an associated photographic image. The attributes (e.g., color) of each pixel of the warped photographic image are derived from at least one pixel of the associated photographic image.
In still another aspect of the invention, a method of stitching together a series of photographic images to build a spherical panorama, includes computing sight directions for each of the photographic images; adjusting the intensity of the photographic images to an intensity related to an average of the intensities of the photographic images; and stitching together contiguous ones of the photographic images.
Embodiments of this aspect of the invention may include one or more of the following features.
Computing sight directions includes performing both vertical and horizontal registration of contiguous photographic images. In one implementation, a correlation-based algorithm is used to compute image phase correlation for all possible positions of image alignment. The position with the minimum alignment difference is then selected. The results of performing horizontal and vertical registration are inspected with a software tool.
Vertical registering is used to derive vertical relationships between images. One approach for vertically registering contiguous photographic images includes rotating each photographic image 90xc2x0; image warping the rotated photographic image with xcfx86L set to 90xc2x0; performing horizontal registration of contiguous photographic images; and then determining sight directions from the image positions after image registration.
Horizontal registration is performed to derive horizontal relationships between images and is usually performed after vertical registration. The method can further include a second xe2x80x9cfine-tuningxe2x80x9d vertical registration.
Image blending of overlapping regions of the contiguous photographic images can also be performed. For example, intensity levels of pixels in the overlapping regions using the following equation:       I    ⁢          (              r        i            )        =                              d          A          t                ·                  I          ⁢                      (                          p              i                        )                              +                        d          B          t                ·                  I          ⁢                      (                          q              i                        )                                              d        A        t            +              d        B        t            
where:
pi is the location of a pixel from a first one of the photographic images;
qi is the location of a pixel from a corresponding second one of the photographic images contiguous with the first one of the photographic images;
dA is the distance between location pi and a boundary of the first one of the photographic images;
dB is the distance between location qi and a boundary of the second one of the photographic images;
I(pi) is the intensity of the pixel at location pi;
I(qi) is the intensity of the pixel at location qi; and
I(ri) is the resultant intensity of the pixel.
When the panorama is navigated, an image browser for viewing spherical panoramas can retrieve the scene image from the spherical environment map based on the desired direction of view.
Other aspects of the invention include systems for implementing the methods described above. In particular, the systems are for photographing pictures of a panoramic scene for use in building a spherical panorama, for transforming photographic images of a panoramic scene into a spherical environment map, and for stitching together a two dimensional array of the photographic images to build a spherical panorama. The systems include computer-readable media having computer instructions for implementing the methods described above.
A computer-readable medium includes any of a wide variety of memory media such as RAM or ROM memory, as well as, external computer-readable media, for example, a computer disk or CD ROM. A computer program may also be downloaded into a computer""s temporary active storage (e.g., RAM, output buffers) over a network. For example, the above-described computer program may be downloaded from a Web site over the Internet into a computer""s memory. Thus, the computer-readable medium of the invention is intended to include the computer""s memory which stores the above-described computer program that is downloaded from a network.
Other advantages and features of the invention will be apparent from the following description and from the claims.